The invention represents a novel method for recording and viewing stereoscopic images by making use of a partially separated encoding of the depth information and the colour values.
The so-called anaglyph method for recording and viewing stereoscopic images has been known for around a hundred and fifty years. In its most common form, as applied to printed images, two mutually extinguishing filters are placed in front of the viewer""s eyes and the stereogram (stereoscopic image pair) to be viewed is printed in two suitably chosen dyes, as two images brought xe2x80x9cin registerxe2x80x9d, i.e. overlapping correctly. The filters and the dyes are complementary in the sense that, ideally, graphics printed in the one dye looks black when viewed through one filter and indistinguishable from the neutral background when viewed through the other and vice versa. Thus, the filters in front of the viewer""s eyes effectively separate the two image impressions in accordance with the stereoscopic principle.
As described above, the method produces monochrome images; and it is in fact quite common to further confine the applications to line drawings and the like, where only purely black lines and curves on a neutral (xe2x80x9cwhitexe2x80x9d) background, or vice versa, are used. When either the filters or the dyes or both fail to meet the ideal requirements, a number of problems can arise, the most typical being that of cross-talk or ghosting (a colloquial term implying xe2x80x9cghost imagingxe2x80x9d), where one eye receives, in addition to its xe2x80x9cownxe2x80x9d image, a faint copy of the image meant for the other eye. In severe cases, ghosting can interfere with stereopsis to the point of complete obstruction, especially in prolonged stereo viewing sessions. (The word diplopia, meaning xe2x80x9cdouble visionxe2x80x9d, is used for this situation; but it must be understood to mean xe2x80x9cfailure to obtain stereoscopic fusionxe2x80x9d, rather than xe2x80x9cobservation of a double imagexe2x80x9d, since the latter can happen in stereoscopy with the perception of parallactic depth still active).
Since its discovery, the anaglyph method has been applied to line graphics, photography, motion pictures, television and computer graphics. Each of these media presents its own version of the problem of non-ideal filters and image colorants.
Quite apart from the efforts towards reducing or eliminating the ghosting in monochrome anaglyphs, several attempts have been made to extend the original method so as to allow it to give the viewer the impression of seeing a stereoscopic colour image.
Like the original anaglyph method these extensions make use of two coloured filters placed in front of the viewer""s eyes. Each of the two images forming the stereoscopic pair is again displayed in colours so chosen that, ideally, each eye receives only its corresponding image, thus allowing stereopsis. However, in order that an impression of a coloured stereo image can arise, the images reaching the eyes after passing the filters can no longer appear as black on a neutral background; indeed, taken together (in the appropriate sense), they must contain all or practically all colours present in the original scene.
In the prior art, the filters chosen are approximations to ideal complementary side band filters or to an ideal mid band filter and its complementary, where the xe2x80x9cbandsxe2x80x9d in question are ranges of the visual spectrum. Also, the two images recorded to form the stereogram are obtained essentially by filtering the original stereo pair according to this ideal of complementarity. If this filtering is performed optically, the partial images are typically passed through the same coloured filters as used for the viewing, either during the recording of the scene or as a post-processing of the images before display; if done electronically, the customary approach is to use the typical RGB-representation of the image, letting one image retain only one of these three primaries, the other only the two remaining. Either way, the two colour-filtrated partial images are then fused into a single stereogram. Thus, all colour anaglyph techniques rely on the socalled tri-stimulus hypothesis, central to colour science, that all perceivable colours within a given colour gamut can be synthesized as a weighted sum of three basic colours, negative weights requiring a special interpretation. The tri-stimulus hypothesis is assumed throughout this document, and a few remarks about halftoning, undercolour removal and black generation will be made below.
The fusion of the received colours and the correlation of the colours with the stereo image takes place in the viewer""s eye-brain system; and in practice, one of the viewer""s eyes will receive a much larger range of hues than the other. In the prior art, it is thus tacitly assumed that the human eye-brain system can synthesize essentially any colour, provided only that one eye receives the partial signal corresponding to the original colour""s content of one primary, while the other eye receives the partial signal corresponding to the colour""s content of the two other primaries.
Prior art techniques may be seen in U.S. Pat. No. 4,134,644 and U.S. Pat. No. 4,247,177.
In addition to ghosting, colour anaglyphs as produced with the prior art have been encumbered by one or more of the following problems, depending on the combinations of filters and display colours used:
The range or xe2x80x9cgamutxe2x80x9d of colours actually perceived by the viewer has been significantly reduced relative to that of the original scene recorded in the stereogram.
Some colours actually perceived by the viewer have shown significant chromatic deviations from the original colours in the scene recorded.
The total amount of light passing through the filters has often been significantly smaller than the light that would have reached the viewer""s eyes, had no filters been applied; and, more importantly, individual colours or colour ranges have shown different degrees of lightness reduction, leading to lightness imbalances in the perceived colours.
The total amounts of light reaching each of the viewer""s eyes have differed significantly, and the difference has not been correlated to luminance differences in the original scene. More precisely: owing to the colour separation, a scene colour may require of the human observer""s eye-brain system an averaging of a component of a relatively high lightness seen in one eye as against a component of a relatively low lightness seen in the other. This in itself is a considerable psycho-physical challenge. If two such scene colours of approximately the same original lightness but different hues are adjacent in the image, yet appear as light and dark in opposite eyes, the two resulting averaging processes are in direct opposition.
Some colours perceived by the viewer have exhibited an undue amount of stereoscopic sheen, a characteristic lustre observable in all kinds of stereograms in homologous areas of different colours. (The strength of sheen depends in a complicated way on the original scene colour, on the colour differences between the homologous areas, on the luminance differences of the homologous areas and on the viewer""s perception).
The basis of the present invention can be demonstrated by two simple experiments:
1) If a stereogram is ideally colour-separated (by electronic means, as described above) and the resulting partial images are displayed side-by-side for optical fusion, using e.g. a Brewster-type stereoscopexe2x80x94instead of overlapping, for fusion by means of coloured viewing filtersxe2x80x94it is easily observed that even in this ideal case, the above-mentioned problems occur, the absolute loss of lightness owing to the absorption in viewing filters excepted, of course. In other words: the human eye-brain system is not fully capable of performing the colour fusion tacitly assumed by the prior art.
2) If, instead, the stereogram is so xe2x80x9ccolour-separatedxe2x80x9d that one partial image retains all colours while the other is converted to a gray-scale image, the latter further subjected to some reduction of contrast and luminance, and the resulting images again presented side-by-side and optically fused, a full-colour stereo image is seen with essentially no visual artefacts added.
The present invention therefore in a sense aims at approximating the latter kind of separation, while still using coloured filters for viewing and a colour separation process in the preparation of images for display, as in conventional anaglyph techniques. In doing so, the invention makes use of a novel combination of viewing filter colours and optionally of a novel colour separation process in the recording or fusion and a colour correction process applied to the stereogram prior to display, so as to allow a very wide range of faithfully reproduced colours, some control of the level of light intensity distributed to the viewer""s eyes over the range of observable colours, a fair elimination of unwanted sheen and a very low level of observed ghosting.
Since no simple optical filter exists which combines the effects of reducing a coloured image to an essentially monochrome image and allowing some kind of xe2x80x9ccomplementaryxe2x80x9d filter to show a full colour image, compromises must be sought. Further experiments with side-by-side mounted stereograms help to establish bounds for, among other things, purely stereoscopic cross-talk, colour cross-talk, contrast and luminance reduction in one image, partial colour reduction in one image and even reduction of sharpness in one image. Such experiments and the resulting bounds form the basis of a test for combinations of filter pairs and display primaries which together will allow the novel implementation of colour-coded stereoscopy.
All conventional information needed about colour science, including standard tables and methods of computation, can be found in good text-books on the subject. (The reader may wish to consult D. L. MacAdam: Color Measurement, Springer-Verlag 1985, D. B. Judd and G. Wyszecki: Color in Business, Science and Industry, Wiley 1975 and the Handbook of Kodak Photographic Filters, Eastman Kodak Company 1990). A shortcut through the theoretical background will be made here by the simple device of a vocabulary, narrowing down the meaning of a number of words as used in this document.
The word colour is used in this document in a number of conceptually slightly different meanings: thus, a dye or colorant, a spectral distribution, a perceived visual attribute that involves a perceived hue etc. all fall under the everyday use of the word xe2x80x9ccolourxe2x80x9d; and where no more specialized term is used in the following, the interpretation will either be obvious from the context or several of the different substitute words will make essentially equal sense, whence the use of a broader term.
For definiteness, it is pointed out that the words monochrome and monochromatic denote the appearance of only one hue, but allow this to be any hue and also allow the full range of saturations from 0% (equivalent to xe2x80x9cblackxe2x80x9d) to 100%. When there is no perceivable (nameable) hue in a monochrome image, it is customarily called a gray-scale image. (When an image is truly black-on-white or any other combination of one specific colour on another, it is usually called a xe2x80x9ctwo-tonexe2x80x9d image.
A spectrum or spectral distribution is, for definiteness, taken to be a set of pairs of real numbers, (w, s), where w lies in the range [400;7001] and s in the range [0;1]. The number w is taken to represent wavelength in nanometers, and the number s some measure of strength, with xe2x80x9c1xe2x80x9d representing some kind of maximum, just as often represented by the equivalent xe2x80x9c100%xe2x80x9d. Ideally, a spectral distribution contains exactly one pair for all possible values of w in the rangexe2x80x94in other words, it forms a continuum. In practice, as always, the spectrum will be a finite set, and xe2x80x9cmissingxe2x80x9d values must be obtained by some choice of interpolation method. If a light can be produced to match, according to some fixed means of measurement, a given spectral distribution, it will in general give rise to a colour perception in a human spectator. xe2x80x9cWhitexe2x80x9d light has an essentially even distribution, i.e. all s-values being essentially the same; but it should be borne in mind that colour adaptation in a human observer allows a wide range of spectral distributions to serve as xe2x80x9cwhitesxe2x80x9d and, accordingly, makes colour matching and the mapping from spectral distributions to e g. (R,G,B)-representations depend on the display set-up.
Metamers (or: metameric colours) are colours which arise from different spectral distributions, but are perceived as identical.
A primary or primary colour is a colour in a finite set, the set of primaries, used for specification of other colours. It is customary, and very wise, to select the primaries in such a fashion that their convex combinations (i.e. positively weighted averages) span the largest obtainable area in the planar model of pure hues, the so-called colour gamut of the display at hand. The most common bases are primary sets of red, green and blue (RGB) for additive displays; and cyan, magenta, yellow (CMY) for xe2x80x9csubtractivexe2x80x9d displays (prints). Note: since the behaviour of additive and subtractive primaries differ in practicexe2x80x94one example is the use of undercolour removal and black generation in prints, leading to the CMYK primary setxe2x80x94all primaries will henceforth be taken as additive, for clarity. When xe2x80x9csubtractivexe2x80x9d displays are to be dealt with, the primaries discussed in the text are additive colours (i.e. spectral distributions) which have been chosen to represent certain spectral distributions reflected off the display surface. These additive colours are almost invariably taken to be essentially complementary to the xe2x80x9csubtractivexe2x80x9d primaries.
To avoid confusion, primaries within a given (full) set of primaries are said to be collateral, the word complementary being reserved for its more intuitive meaning as applying to hues, spectral distributions, value sets etc. In colour theory based on the CIE chromaticity diagram, the word complementary can be given a precise meaning when applied to spectral colours; but usually, the intuitive meaning suffices.
A colour attribute is here an attribute of a colour, rather than of an object. Although perceived colours are the results of purely mental activities in the human observer and perceived colour attributes in that sense independent of the original cause of the colour signal, the names and characteristics of colour attributes (e.g. xe2x80x9cred contentxe2x80x9d, xe2x80x9csaturationxe2x80x9d, xe2x80x9cshadexe2x80x9d) usually make sense only within a specific colour representation scheme.
A filter is a means of changing the colour attributes of a given (range of) colour(s). It will be assumed that the effect of any filter can be prescribed as a mapping from S to itself, where S denotes the set of all spectral distributions of relevance in the context. The absorption of coloured optical filters is thus modelled by the spectral distribution obtained when white light is passed through them.
An ideal filter is a filter which acts on all spectral distributions with mathematical exactness in accordance with its definition. An approximate filter acts in accordance with its definition to within an error margin which may be explicitly stated or implicit in the description. Real-life transmission or reflection filters are approximate, filters in mathematical models or in computations acting on integers are ideal. The word filter alone will cover both meanings, but is usually intended to signify an approximate filter.
A subtractive filter acts on a given spectral distribution by replacing every s-value in the distribution by a value which is not larger than the original. An additive filter may replace some s-values in a spectral distribution by a higher value. The meaning of the words subtractive and additive as applied to filters should not be confused with the commonly used meaning, describing two modes of colour blending, namely xe2x80x9cpaint mixingxe2x80x9d, which is subtractive insofar as it is absorptive, and xe2x80x9clight mixingxe2x80x9d. Also, note: additive filtering can not be achieved optically, since all optical filters absorb a certain amount of the light passing through them. It can be emulated optically by the combined use of a subtractive filter and increased light intensity; and it can be achieved computationally, in a model of the filtering process.
Given the spectral distributions of a set of primaries and a chosen primary from the set, an associated 1-primary filter is a filter which reduces any s-value in an arbitrary spectral distribution to at most the corresponding s-value of the primary. A 2-primary filter reduces all s-values of an arbitrary spectral distribution to at most the highest of the corresponding s-values of two given primaries.
To make certain descriptions in the following short and clear, let a 0-primary filter be a filter which reduces all s-values to their luminance content according to a chosen luminance measure, optionally scaled by a common factor. Optically, such a filter can only be approximated, although its effect can be emulated almost to perfection by photographic means. Electronically and computationally, a 0-primary filter can be realized to the accuracy allowed by the electronic equipment and the computational representation, respectively.
For completeness, a 3-primary filter is a filter with no effect other than a possible scaling by a common factor on the (relative) spectral distribution of any colour. Here, the tri-stimulus hypothesis is clearly at work, but the phrase xe2x80x9c3-primary filterxe2x80x9d should not be taken to imply that the full set of primaries has exactly 3 members, although this will be the case in most practical applications.
A combination (of) filter(s) is a filter with the combined effect of two or more filters. In computational emulation of filters, the combined effect is often modelled using a pointwise product.
A null filter excludes all visible light, i.e. regardless of the spectral distribution, all s-values are set to zero by a null filter.
A neutral density filter has the same relative effect on all s-values in a spectral distribution. This definition makes the phrases xe2x80x9cneutral density filterxe2x80x9d and 3-primary filter synonymous.
A (ideal) partitioning filter set is here meant to be a filter set (usually a pair) so specified, that: 1) any combination of a pair of different filters from the set is a null filter; and 2) to any (w, s)-pair, a filter in the set can be found which has no effect, or the same effect as a neutral density filter, on the given (w, s)-pair.
A (1,2)-partitioning filter pair is here meant to be a partitioning filter set consisting of two filters, so adapted to a pre-selected set of primaries that one filter (the xe2x80x9c1-primary filterxe2x80x9d or xe2x80x9c1-filterxe2x80x9d) acts towards one primary as if no filter, or a neutral density filter, was applied, the other (the xe2x80x9c2-primary filterxe2x80x9d or xe2x80x9c2-filterxe2x80x9d) acts towards the remaining primaries as if no filter, or a neutral density filter, was applied. By their partitioning property, the 1-filter will, generally speaking, exclude light from the two collateral primaries and vice versa, although if the primaries are determined by colour matching, the effect of metamerism may make this relation much more complicated.
A leaky (1,2)-partitioning filter pair is here meant to be an approximate (1,2)-partitioning filter pair in which the 1-filter approximates the ideal, i.e. as far as possible allows passage of all light stemming from one primary and of none from the two remaining primaries; while the 2-filter allows passage of a certain amount (the leak) of light from the one primary and as far as possible all light from the remaining two primaries.
The stereoscopic principle as applied to the recording and stereoscopic display of a stereogram showing a 3-dimensional scene is assumed familiar. It is worth mentioning that a conventional side-by-side or two-channel stereogram consists of two images, called xe2x80x9cpartial imagesxe2x80x9d in the following (although they are usually only xe2x80x9cpartialxe2x80x9d when considered parts of the stereogram); and that two points, one from each partial image, both being images of the same point in the 3-dimensional scene, are said to be homologous. In side-by-side stereoscopy, the fusion is usually performed by means of prismatic lenses. More generally, two-channel stereoscopy keeps the partial images separate in the display device, but perform a fusion into one image seen by the human observer.
The general nature of the invention can be stated using some of the definitions given above.
The present inventors have identified the problems of the prior art of colour anaglyphs to arise from two distinct but connected sources:
In the prior art, the use of partitioning viewing filter pairs typically makes it difficult for the human spectator""s eye-brain system to synthesize a full colour gamut. Therefore, the viewing filter pair must be a suitably chosen leaky (1,2)-partitoning pair, where it turns out that the allowable leak is of a special nature and that certain parameters, characteristic of such a filter pair, lie in a narrow range. The special recording, separation and display techniques required to allow the use of such filters will henceforth be called colour coded stereoscopy.
In the prior art, the deviations of perceived colours from intended or recorded colours fall in separately identifiable, but not mutually exclusive, categories. All categories can be addressed; but it should be noted that conventional colour management techniques serve as intermediates between recorded and displayed colours, and do not address the special anomalies of the final, perceived colours in a colour anaglyph or a colour coded stereogram. Therefore, the colour correction to be performed (often as a post-processing) on a colour anaglyph or a colour coded stereogram can advantageously make use of special adaptations of colour management techniques. This kind of processing, applied during or after the recording of a colour coded stereogram, will be called colour correction.
Since the problems of choice of filters and of colour correction (for each and all of the categories of colour deviations) are interconnected, the various aspects of the invention address these problems as a whole, where possible, but otherwise from whatever angle an improvement is feasible, given that not all parameters are variable in all practical circumstances.
The detailed description of the invention falls naturally in three parts: a description of the method of choosing filter pairs, including a qualitative description of some of the specific filter pairs that appear in the embodiments and the claims; a description of the choices of methods for obtaining a colour coded stereogram or, in the case of two-channel imaging, approximating colour coding by novel separation techniques; and a description of the colour management techniques which are preferably applied to identify and adjust the problematic colour ranges described by the invention in order to mitigate their effects on the final perceived colour coded stereogram.
For most display purposes, colour mixing is additive and the primaries accordingly red, green and blue (R, G and B). For printing, colour mixture is essentially subtractive (see below) and the primaries used are therefore cyan, magenta and yellow (C, M and Y), with black added for practical purposes. In specification of filters for colour coded stereograms, the 1-primary must be taken from the (R,G,B)-set, since the colour mixture is eventually based on light reaching the observer""s eyes and therefore essentially additive. Addition of equal amounts of the two collateral primaries produce the complementary xe2x80x9csubtractivexe2x80x9d primary. A simple notation for the basic colours of the (1,2)-partitioning filter pairs is therefore (R,C), (G,M) and (B,Y).
It should be noted that the visual appearance of the filters actually used may not correspond to the naive interpretations of these letters. In particular, some C-filters will appear xe2x80x9cbluexe2x80x9d to most observers and some Y-filters xe2x80x9corangexe2x80x9d or even xe2x80x9cbrownxe2x80x9d, and these are not normally considered as primary colours.
Even the choice of basic colours of the (1,2)-partitioning filter pair depends on the intended use, a rule of thumb being as follows: (B,Y)-filters generally approximate the ideal (0,3)-partitioning best in the sense that Y-filters let through a wide range of (perceived) colours and B-filters give a nearly monochrome reduced-contrast image. Also, (B,Y)-filters are relatively robust to variations in filter production quality and in display conditions. However, (B,Y)-filters may require a strong colour adjustment; and they often give a substantial lightness imbalance between the images reaching the observer""s eyes. (R,C)-filters usually require less colour adjustment and have a better lightness balance, but are more sensitive to variations in display xe2x80x9ctemperaturexe2x80x9d (the equivalent black-body temperature of displayed xe2x80x9cwhitexe2x80x9d). (G,M)-filters generally have the best lightness balance and in principle require the least colour adjustment; but they give rise to a noticeable sheen ranging over almost all perceivable hues. Also, the fact that the human eye-brain system has its highest resolution in the hue area of green seems to make (G,M)-filtration very sensitive to errors or deviations. Clearly, the importance of each of these various phenomena varies with e.g. the basic colour temperature of the display medium, ambient light in the display set-up, duration and intensity of observation, colour range in images displayed etc., etc. The invention allows the best possible compromise to be sought.
The process of finding a suitable filter pair can be wholly experimental, wholly computational, or a mixture, and can be iterated towards a final optimum as many times as desired; but the following test excludes all irrelevant filter combinations in a very simple fashion:
Assume a candidate filter pair is to be tested. Call the filters xe2x80x9cthe 1-filterxe2x80x9d and xe2x80x9cthe 2-filterxe2x80x9d as described above. With the usual tri-stimulus primaries (R,G,B), one of the partitions R vs. G-and-B, or G vs. B-and-R, or B vs. R-and-G would then be tested. More generally, assume the display primary set P is given and partitioned into two disjoint sets I and II, i.e. P=I+II and I and II have no common member. An additive display device is then chosen, essentially identical to the intended final device. If the final display is subtractive (e.g., graphical or photographic prints), the colours intended as additive display primaries are chosen, the spectral properties of their printed equivalents may be measured, and the test carried out using an additive display device and additive primaries having been chosen to represent their printed counterparts. The major reasons for this circumspect approach are two: the viewing filters will eventually act on light that reaches the observer""s eyes; and the complications of printing, especially half tone printing, makes it difficult to assign a single spectral distribution to a printed display colour, even when the light and printing base are known.
Let the phrase xe2x80x9cmeasuring the y-valuexe2x80x9d be a shorthand notation for xe2x80x9cmeasuring the spectral distribution and computing the y-value according to the 1931 CIE colour matching specificationxe2x80x9d. This can be done in a standardized fashion. Then the test consists of:
Step 1: Display a uniformly coloured image in the most luminous colour (usually xe2x80x9cwhitexe2x80x9d) available on the display device, when using the full set of primaries chosen. Measure the y-value of this image as seen through the 2-filter and call the result Lu.
Step 2: Similarly, display the most luminous colour available on the display device when using colours from the I-set. Measure the y-value of this image as seen through the 1-filter and call the result L11.
Step 3: Measure the y-value of the image from step 2 as seen through the 2-filter and call the result L12.
Step 4: Again similarly to step 1, display the most luminous colour available on the display device when using the II-set. Measure the y-value of this image as seen through the 1-filter and call the result L21.
Step 5: Measure the y-value of the image from step 4 as seen through the 2-filter and call the result L22.
Step 6: Compute the numbers R1=L21/L11, R21=L21/Lu, R2=L12/L22. Call the numbers R1, R21 and R2 the result of the test.
A few brief comments are necessary at this point:
The number R2 essentially measures the leak and hence the possibility of observing the full colour gamut in the stereogram. The number R21 measures the ability of the 1-filter to exclude the primaries belonging to the complementary image. Hence, R21 is a measure of the visibility of the 1-image over any xe2x80x9cghostxe2x80x9d of the 2-image seen through the 1-filter. The bound on R21 given below is fairly tight and in a sense expresses the monochromaticity of the 1-image. The number R1 measures the visibility of the image seen through the 1-filter over and above any ghosting. The test thus offers measures of stereoscopic cross talk and colour leakage that will indicate whether and to what extent colour coded stereoscopy can be obtained. To be useful for viewing colour coded stereograms, a viewing filter pair will generally, according to the invention, have to meet the following conditions:
Condition 1: R2xe2x89xa70.005
Condition 2: R21xe2x89xa60.01
Condition 3: R1xe2x89xa60.05
Note 1: Strictly speaking, there should be an upper bound on R2, expressing the xe2x80x9cno cross talkxe2x80x9d upper limit; but this is a general stereoscopic condition, rather than a condition associated with the stereoscopic encoding method of the invention.
Note 2: A viewing filter pair that performs well with one set of primaries may perform badly with another and even fail to pass the above test, often a consequence of differing display device xe2x80x9ccolour temperaturesxe2x80x9d; and a filter pair that passes the test with good result numbers may still require a more substantial colour correction and hence be less useful than a filter pair that barely makes it. For some purposes, the requirements on R2 may actually be stronger than the limit stated above.
Note 3: In some embodiments of the invention, use is made of separation filters placed before the partial images in a two-channel set-up, as an emulation of colour coding proper. The test then requires that the primaries be selected and passed through the appropriate separation filters before the y-measurements. The reason for this is that the test monitors the behaviour of the viewing filters vis-a-vis the displayed spectral distributions.
Image separation in the prior art has usually been straightforward: one xe2x80x9ccolour planexe2x80x9d or xe2x80x9ccolour channelxe2x80x9d was removed from one of the partial images of the stereogram and replaced by the corresponding colour plane from the other partial image. This gives essentially the right colours in the anaglyph as seen without filters and may also give the correct stereoscopic effect as seen through the filters. However, as noted above, the perceived colours in the stereogram will fail when conventional viewing filters are applied. Moreover, the stereoscopic effect will suffer under subtler conditions: consider e.g. the use of a (B,Y)-viewing filter pair and a stereogram showing a red flower on a background of green foliage. The actual graphical content of the blue planes of the partial images of the stereogram will be confined to highlights in the picture, which means that conventional anaglyph separation leaves one partial image almost black. As seen through the viewing filters, therefore, the final image may have very little observable depth information.
On the other hand, using the conventional gray scale image corresponding to one partial image to replace the blue plane of the other (e.g. electronically by treating the gray scale image as one colour plane, possibly by converting it to xe2x80x9cfull colour representationxe2x80x9d and taking out the blue plane) will grossly deform the observable hue circle, although on the average an equal number of hues will receive too little or too much blue.
The ideal is still to approximate a (0,3)-representation, and therefore the image separation should use a weighted average somewhere between the two extremes. In the (B,Y)-case, using the blue plane alone corresponds to using the weights (0,0,1) applied to the (R,G,B)-primaries, while the grays are typically given by the NTSC-luminance weights (0.3, 0.59, 0.11). Possible alternatives in the (B,Y)-case would be (0.11, 0.22, 0.67) or (0.17, 0.17, 0.66), giving natural looking blue colours (as seen without filters as well as through the filters) except in isolated xe2x80x9cdeep bluexe2x80x9d areas, yet salvaging depth information from the other two colour planes. Note, that this kind of image separation has part of the colour correction built into it.
The xe2x80x9cfull colourxe2x80x9d image to be seen through the leaky 2-filter may keep as large an amount of its third colour as is commensurate with the cross-talk limits used in the test. Alternatively, all of the third colour is removed and spot colour correction performed after image fusion. For the best overall result, colour correction as described in the next section is recommended.
In a related, but different technique, these essentially one-channel images are shown on two-channel displays, for instance by means of a conventional stereo projector or two standard projectors producing overlaid images on a screen. Then the colour separation can be performed optically as part of the display process, and colour separation filters can be used, which give related weights to the primaries, although an optical filter can not shift luminous energy from one spectral range to another. The colour separation filters must be chosen with the spectral properties of the display device and of the viewing filter pair in mind, orxe2x80x94much betterxe2x80x94the separation filter pair, the display device and the viewing filter pair must be chosen together, i.e. optimized according to the invention. In particular, the separation filters are preferably xe2x80x9cwhite balancedxe2x80x9d, i.e. when white light is shone through the two filters and the resulting colours added on the display, the visual impression must again be xe2x80x9cwhitexe2x80x9d. This is not required, and rarely achieved, in the viewing filters.
As mentioned above, the use of colour correction may often be restricted to specific image areas or narrow hue ranges and serve to alleviate the few adverse effects of colour coding. Quite apart from spot colour deviations, a general drift of hues will usually be observed in an uncorrected stereogram. Correction for the hue changes, along with correction aimed at the problems mentioned in the introduction, can be performed at all levels: general hues (even colour tables), full colour space, individual pixels in the fused image, individual pixels compared with original pixel values, individual pixels compared with homologous point pairs andxe2x80x94going to the absolute extremexe2x80x94lists of pixels in which each but the first is the observed homologous point belonging to its predecessor on the list.
Colour computations and colour match experiments are described in detail in the books mentioned above and their references; so to illustrate their use in the context of the present invention it suffices to show how to identify and address the phenomena relevant to colour coded stereograms. For the sake of conciseness, almost all operations in the following will therefore be linear and based on a relatively low spectral resolution (one reading per 25 nanometers in the range 400 to 700 nanometers).
So, assume for definiteness that a display method is given which makes use of RGB-primaries corresponding to xe2x80x9cwhitexe2x80x9d light at 6500 K, as computed according to the CIE formulas, then filtered through the Kodak Wratten (R,G,B)-separation filters. Likewise, assume that the given viewing filter pair are a blue and a yellow, respectively, with the spectral transmittance effects listed in Table 1 (in the range from 400 nm to 700 nm, steps of 25 nm, values recorded to two decimal digits).
Assume that the perceived colour of a single human eye can be modelled by RGB-values computed by means of the xe2x80x9ceyexe2x80x9d recommended by the Optical Society of America:
R=0.799X+0.4194Yxe2x88x920.1648Z
G=xe2x88x920.4493X+1.3265Y+0.0927Z
B=xe2x88x920.1149X+0.3394Y+0.717Z
where X, Y and Z are the 1964 CIE tristimulus colour values at 6500xe2x80x3 K, again taking the 13 values in the range 400 nm to 700 nm, steps of 25 nm.
To make the eye and the primitives xe2x80x9chue consistentxe2x80x9d, introduce a calibration: the (R,G,B)-to-(R,G,B) transmission will in the present approximation be linear, given by the matrix M=Exc2x7(L::[C;C;C])T, where E is the 3 by 13 matrix of eye responses, L is the 3 by 13 matrix of primary spectral values and the operation L::[C;C;C] corresponds to the action of L on the spectral distribution of 6500K white lightxe2x80x94in essence a pointwise multiplication of spectral transmission values. Ideally, the resulting matrix M should turn out to be the identity matrix, but this requires negative lobes in either the primitives or the eye, or both. Ideal hue consistency (no hue drift) requires M to be a positive definite, symmetric circulant matrix (rows of M having the form a b b, b a b, b b a with axe2x89xa7l/3); and approximate hue consistency (uniform hue drift) requires that M be circulant (rows of M having the form a b c, c a b, b c a). Introduce two diagonal matrices, D and H, and replace M by Dxc2x7Mxc2x7H, the equivalent of calibrating the settings of the primaries according to H and of the eye readings according to D. Determining D and H by performing a non-linear least squares optimization (keeping their first elements identically equal to 1 for definiteness) towards the goal that Dxc2x7Mxc2x7H be circulant results in the calibration factors: (1, 1.64, 1.044) as diagonal entries of D, and (1, 0.593, 1.017) as the diagonal entries for H. For definiteness, Dxc2x7Mxc2x7H can be finally subjected to a scaling, so that (R,G,B)=(1,1,1) maps to a vector with maximum element 1.
As an auxiliary formula for the gray level of an (R,G,B)-colour, take the Y-value (also sometimes called luminance) of the official NTSC formula; i.e. if r is a vector of (R,G,B)-values, let the scalar product y=[0.30 0.59 0.11]xc2x7r represent its NTSC-luminance. The operational set of colours will be the (R,G,B)-values for 36 colours of equidistributed hue, maximal saturation and maximal value, according to the xe2x80x9chex-conexe2x80x9d HSV model. The following calculations will be done using (R,G,B)-values and the compound matrices as described above.
The model for the colours seen by one eye through filter F1 is summarized in a matrix M1=Dxc2x7Exc2x7(L::[F1::C;F1::C;F1::C])Txc2x7H, where again the symbol:: denotes pointwise multiplication, the computational equivalent of filter combination. By the same token, the effect of F2 is given by M2=Dxc2x7Exc2x7(L::[F2::C;F2::C;F2::C])Txc2x7H. The row vectors F1 and F2 hold the rows marked xe2x80x9cBxe2x80x9d and xe2x80x9cYxe2x80x9d in Table 1, respectively. Then the displayed-to-perceived (R,G,B) transmission can be modelled by the weighted average P=wM1+(1xe2x88x92w)M2. For the present purposes, take w=l/5 to obtain xe2x80x9cwhite adaptationxe2x80x9d very much like that of the true human eye-brain system. Much more sophisticated (non-linear) models of the perceived colours are possible and often useful; but the colour correction actions described below remain qualitatively the same. Finally, to avoid comparisons of e.g. lightnesses perceived with and without viewing filters, scale P so that (again) (R,G,B)=(1,1,1) maps to a vector with maximum element 1.
The resulting matrices are:       M    =                            0.7265                          0.2351                          0.0064                                      0.2444                          0.6504                          0.1052                                      0.0398                          0.1333                          0.6950                                M      1        =                            0                          0.0000                          0.0025                                      0                          0.0003                          0.0346                                      0                          0.0005                          0.2924                                M      2        =                            0.0939                          0.0235                          0.0002                                      0.0310                          0.0611                          0.0046                                      0.0051                          0.0121                          0.0259                          P    =                            0.7937                          0.1989                          0.0073                                      0.2624                          0.5174                          0.1120                                      0.0428                          0.1033                          0.8370                    
A perceived colour (where the viewer has the filter F1 in front of one eye and the filter F2 in front of the other), as computed according to this discrete, piecewise linear model, is now given as p=Pr, where r is the vector of displayed (R,G,B)-values, and p the vector of perceived (R,G,B)-values.
The first step of the general colour correction of a colour coded stereogram consists of a shift of all hues, in order to balance the hue deviations. For the present example, a shift corresponding to an addition of l/144 to all hues give a better distribution of hue deviations, as can be seen in Table 2: With the displayed hues given in the rows marked xe2x80x9cHxe2x80x9d and the perceived hues in the rows marked xe2x80x9cpxe2x80x9d, the largest deviations are found in the xe2x80x9cG to Cxe2x80x9d range, column 5 (0.4571 vs. 0.5252) and in the xe2x80x9cM to Rxe2x80x9d range, column 1 (0.8325 vs. 0.8074), the difference range thus being [xe2x88x920.0681; 0.0251]. After the shift, the perceived hues are as in the rows marked xe2x80x9cPsxe2x80x9d, and the largest deviations are now balanced, as seen in the xe2x80x9cG to Cxe2x80x9d range, column 6 (0.4914 vs. 0.5410) and in the xe2x80x9cM to Rxe2x80x9d range, column 2 (0.8804 vs. 0.8298), the difference range being the nearly symmetric interval [xe2x88x920.0496; 0.0506 ].
After the hue shift, there will be at least one, and generally an even number of hues that are truthfully rendered, the fixed points of the hue-to-hue mapping. Between such fixed points, the hue circle can now be further re-parameterized to give a more truthful rendition of all colours. From a mathematical point of view, such a re-parameterization has the effect of a local inversion, and so can be done by, say, interpolation. (For ease of comparison of values, no such re-parameterization is done here).
Swoon colours are those for which there is a severe lightness loss from displayed colours to perceived colours. Flare colours, conversely, are perceived as being too luminous. The simple remedy is to adjust the lightness of all hues, for which the perceived lightnesses fall outside some chosen limits. (Where the displayed colours are fully saturated, lightness can only be increased at the loss of some saturation). The ratio of NTSC-luminances of p=Pr to r for the 36 r""s are given in Table 3. In the present case, relatively mild swoon effects are noticeable in the green-to-cyan range, and there is an isolated flare around pure blue. It should be noted, that the latter is automatically reduced by the colour coding technique described above under xe2x80x9cimage separationxe2x80x9d.
The remaining phenomena are best treated as xe2x80x9cspot coloursxe2x80x9d, and hence should only be identified at this stage: Luminance sheen colours are those for which the lightness differences between the colours reaching the two eyes is excessive. Chromatic sheen colours are those for which the differences between the hues of colours reaching the two eyes is excessive. As a rule of thumb, luminance sheen is only really noticeable when chromatic sheen is low and vice versa. A table such as Table 4 helps locating the relevant colour ranges: where one of the numbers marked xe2x80x9cLxe2x80x9d and xe2x80x9cHxe2x80x9d is numerically large and the other small, sheen can be observed. In the present case, there is some luminance sheen around pure blue, which like the flare is alleviated by the colour separation. Chromatic sheen can be observed around pure cyan and pure magenta. Generally speaking, the remedies are to exchange saturation for lightness to alleviate chromatic sheen and vice versa to alleviate luminance sheen. Also, a localized hue shift towards hues with lower sheen effects is allowable for spot colours, and this latter would probably be recommendable in the present case for the chromatic sheens in pure cyan and pure magenta, if they were to appear as intense spot colours in an image.
Finally, ghost colours can be determined by computing the two colours resulting from separation and subsequent passage through one viewing filter, then finding those which give rise to excessive differences when reaching the corresponding eye. Ghost colours must be found for the eyes separately, and as for sheen, a distinction may be made between luminance ghosting and chromatic ghosting. Again, the typical remedy is to allow certain hue shifts; but with ghost colours it should be kept in mind that the effect is typically most visible near the borders of scene objects, so the application of such shifts can be localized to such areas of the image, especially if the hue shift can be made gradual. At the same time, the colour leakage, required as part of the invention, is really a faint (and controlled) chromatic ghost effect and must be identified as such.
xe2x80x9cSpot colourxe2x80x9d correction thus usually takes the form of a combined hue shift and lightness re-balancing between the two partial images, either before fusion or in the fused image.
In a first aspect, the invention is therefore concerned with a test of or a method of identifying a given or proposed viewing filter set as against a given or proposed set of display primaries. The outcome of the test is three numbers.
In a second aspect, the invention is concerned with the application of special colour correction techniques to colour separated stereograms or conventional colour anaglyph stereograms, such that the resulting perceived stereogram has colours close to the those of the original or intended scene, while artefacts caused by the use of coloured viewing filters are reduced.
In a third aspect, the invention is concerned with a method for obtaining a stereogram, called a colour coded stereogram, which consists of two partial images fused together in such a manner that when the stereogram is displayed on a device with known spectral properties and viewed through correctly chosen coloured filter pairs, the effect approximates that of viewing a two-channel display of stereogram consisting of one full colour partial image combined with one gray scale partial image, these two partial images being brought in visual balance.
In a fourth aspect, the invention is concerned with special combinations of coloured filter pairs for viewing colour coded stereograms as well as for viewing conventionally separated colour anaglyphs.
In a fifth aspect, the invention is concerned with a method for choosing or constructing special combinations of coloured filter pairs and optionally combining the resulting choice or construction with the specification of a sequence of colour correction steps to apply to the stereograms to be viewed with the filter pair.
In sixth aspect, the invention is concerned with the application of coloured filter pairs for modifying each of the partial images in a conventional two-channel stereoscopic display, prior to the fusion of the images on the display surface, the coloured filter pairs being chosen so that the resulting fused stereograms are suitable for viewing on the given display with a selected pair of coloured viewing filters, such as the filters mentioned above as a fourth aspect, or filters selected according to the method mentioned above as a fifth aspect.
In a seventh aspect, the method is concerned with the display, using 2-channel techniques, of a stereogram so prepared that one partial image is in full colour, the other monochrome, the visual balance between the two being suitably adjusted.
In an eighth aspect, the invention is concerned with the mounting of the special viewing filter pairs, mentioned above as a fourth aspect, in spectacles, lorgnettes, frames or the like.
In a ninth aspect, the invention is concerned with an apparatus, such as a specially constructed electronic camera, optionally containing a coloured filter pair and containing means of separation and fusion so chosen that images recorded with the camera consist of two stereoscopically fused partial images, the colour contents of which in essence make the picture a colour coded stereogram or a specially prepared anaglyph, suitable for viewing on a given display through selected viewing filter pairs, such as the filters mentioned above as a fourth aspect, or filters selected according to the method mentioned above as a fifth aspect.
In a tenth aspect, the invention is concerned with an apparatus, such as a specially constructed optical adaptor to mount on an existing camera, the adaptor containing a coloured filter pair and means of separation and fusion so chosen that images recorded through the adaptor consist of two stereoscopically fused partial images, the colour contents of which in essence make the picture a colour coded stereogram or a specially prepared anaglyph, suitable for viewing on a given display through selected viewing filter pairs, such as the filters mentioned above as a fourth aspect, or filters selected according to the method mentioned above as a fifth aspect.
Thus, in a first embodiment, the invention makes use of a spectrophotometer for measuring the spectral distributions of filters combined with calorimetric computations of the passage of light of a given spectral distribution through filters of a given spectral characteristic, the computations allowing the numerical determination of the numbers R2, R21 and R1 as described above.
In a second embodiment, the colour deficiences are computed from the known spectral properties of the display primaries and the chosen filters, the computations optionally supplemented or replaced by observations on a series of test images, whereupon a hue shift, a hue re-parameterization, a saturation enhancement distribution and a brightness enhancement distribution are determined from formulas akin to those presented above and applied to electronic images on the picture element level, and methods and criteria for the further application of spot colour correction are chosen and applied.
In a third embodiment, the two partial images of a conventional stereogram are brought into some form of digital representation, the left partial image has its blue xe2x80x9cplanexe2x80x9d or xe2x80x9cchannelxe2x80x9d removed, the right partial image is replaced by a monochrome image resulting as a weighted average of its R-, G- and B-planes with weights (0.15, 0.15, 0.7), and the monochrome image, considered as a one-plane image, is then substituted for the missing plane of the left partial image, see FIG. 18.
In a fourth embodiment, the invention comprises a pair of coloured filters with the spectral transmission characteristics shown in FIG. 2, mounted in a pair of spectacles or a viewing frame for the viewing of colour coded stereograms or conventionally separated colour anaglyphs.
In a fifth embodiment, the invention makes use of known spectral distributions of primaries of a display and selects, from a set of available filters, those pairs which pass the colour coding test as described above and further selects, from the pairs thus found, that pair or those pairs, which give the most satisfactory colour balance, as determined by one or more human observers viewing a selection of test images.
In a sixth embodiment, a stereoscopic image is shown on a screen using a conventional two-channel stereoscopic projector, the partial images being filtered through two coloured filters with spectral transmission characteristics as in FIG. 9.
In a seventh embodiment, a stereoscopic image is shown on a monitor or screen, the partial images alternating (-they are thus stereoscopically separated by means of a display-synchronized viewing device-), one partial image being shown in essentially full colour, the other as a monochrome image, optionally with reduced contrast and intensity.
In an eighth embodiment, filters like those of the fourth embodiment or selected according to the fifth embodiment are mounted in a pair of spectacles.
In a ninth embodiment, two electronic cameras are mounted on e.g. a rail to facilitate variation of the distance of their lenses, the lenses always remaining in the same plane, one or both camera lenses being optionally equipped with coloured filters, the electronic signals from the cameras being subsequently fused, according to the techniques of the third embodiment, into the signal required for the display of one stereoscopic image.
In a tenth embodiment, an adaptor much like a conventional range finder is equipped with separation filters, tested against a pair of viewing filters, such as the filters mentioned above as a fourth aspect, or filters selected according to the method mentioned above as a fifth aspect, and placed in a house that can be mounted on a camera lens, see FIG. 17.
Table 1 shows the spectral transmittances of two coloured filters marked xe2x80x9cBxe2x80x9d and xe2x80x9cYxe2x80x9d in the spectral range from 400 nm to 700 nm, one reading per 25 nm, maximal theoretical transmittance scaled to 1, readings given with two decimal digits.
Table 2 shows the hues according to the xe2x80x9chex-conexe2x80x9d model of the computed (R,G,B) values of 36 colours. The rows marked xe2x80x9cHxe2x80x9d stem from multiples of the form Mr, the rows marked xe2x80x9cPxe2x80x9d from multiples of the form Pr and the rows marked xe2x80x9cPsxe2x80x9d from multiples of the form Ps. Here, M and P are the matrices defined under xe2x80x9ccolour correctionxe2x80x9d above; each vector r is taken from 36 hue-equidistant vectors on the perimeter of the xe2x80x9chex-conexe2x80x9d with the first vector at pure red; and each vector s likewise taken from the perimeter of the xe2x80x9chex-conexe2x80x9d but with l/144 added to the hue-values, as compared to those of the r""s. Table 3 shows ratios of NTSC-luminances, Lm/Lp. The Lm-values were computed from the Mr""s, the Lp-values from the Ps""s, see description of Table 2 above.
Table 4 shows NTSC-luminance differences (rows marked xe2x80x9cLxe2x80x9d) vs. scaled hue differences (rows marked xe2x80x9cHxe2x80x9d) between the (R,G,B)-values computed for each eye, according to the models described under xe2x80x9ccolour correctionxe2x80x9d above. The luminance differences are thus confined to the interval [xe2x88x921;1]. The hue differences were computed modulo 0.5, then scaled by a factor 2 to let them fall in the interval [0;1], facilitating observations.